Show notebooks in Drive CIFAR-100.ipynb - Colaboratory
Google Account
Yonathan Sachs
sachsy@post.bgu.ac.il
Code
Insert code cell below
Ctrl+M B
Text
Add text cell
Toggle header visibility

Assignment 1 | CIFAR-100


Imports and environment setup


[ ]

Set Neptune AI


[ ]

[ ]

[ ]

Neptune example


epochs = 10
batch_size = 128
lr = 0.0001
optimizer = Adam(lr = lr)
loss = "binary_crossentropy"
 
PARAMS = {
    'epochs' : epochs,
    'batch_size' : batch_size,
    'lr' : lr,
    'optimizer' : optimizer,
    'loss' : loss,
}
tags = []
 
 
 
with neptune.create_experiment( name='VGG19',
                                params=PARAMS,
                                description='Using VGG19 model and weights to predict CIFAR-100',
                                tags=tags,
                                upload_source_files=[]) as npt_exp:
 
    callback = [
        # tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.005, patience=5, verbose=0, restore_best_weights=False)
        ModelCheckpoint('best_model.h5', monitor='val_loss', save_best_only=True),
        NeptuneMonitor(npt_exp),
    ]
 
    X_train, X_test, Y_train, Y_test = [],[],[],[]
 
    model = Model()
    model.compile(loss=loss,optimizer=optimizer, metrics=['accuracy'])
    history = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), batch_size=batch_size,
                        epochs=epochs,
                        verbose=1, callbacks=callback)
 
    results = model.evaluate(X_test, Y_test, batch_size=batch_size)
    names = model.metrics_names
    npt_exp.set_property(names[0], results[0])
    npt_exp.set_property(names[1], results[1])
 
    model.save_weights('model_weights.h5')
    npt_exp.log_artifact('model_weights.h5')
    npt_exp.log_artifact('best_model.h5')

Load datasets


[ ]

[ ]

[ ]

1. Data exploration


[ ]

a. Data size


As we can see above, Train set has 50,000 records and the Test set has 10,000 records.

Number of unique targets is 100


b. What data does each sample contain?


Each sample contains a simgle Image and its classification


[ ]

Let's take a look at some examples from the Trainset


[ ]

Dimensions, channels, how many classes


  • Dimensions: (32x32) - each image has 32x32 pixels.
  • Channels: 3 that represents the RGB colors
  • Number of classes: As mentioned before, the num of classes is 100

Should we preprocess the data?


We are using data from the Keras.datasets, thus the data is ready for use.


Can we use augmentation? what kind would be valid?


We can use augmentation, rotations, movements and croping that can be applied here.

But as we can see, the given images are well croped and the main element is centered therefor there is no need for augmentation for this dataset.

Edit - Later while working on the model's improvements, found out that augmentation is helpful (more details at 2.d)


c. Class distribution - is it balanced?


[ ]

The data is perfectly balanced - Each target has the exact same number of records ( 500 for the train, 100 for the test )


[ ]

[ ]

d. Are there any benchmark results for different methods used on this data?


image.png


image.png


We can see that the top 4 methods were able to get an accuracy score above 90%

Note that most of them were using 'Extra training data', this was not common in previous methods ( from #6 downwards )


e. Show some samples for each lable ( easy vs hard )


Hard separables might be caused by similar backgrounds and\or colors of elements in the pictures, as well as similar shapes of the objects.

Easy separables can be seen as images with different structures ( colors, texture, backgrounds)


image.png


2. Build model


[ ]

[ ]

Save and Load models


[ ]

BuildModel


Input shape

  4D tensor with shape:
  if `data_format` is `"channels_first"` - `(batch, channels, rows, cols)`
  if `data_format` is `"channels_last"` - `(batch, rows, cols, channels)`

[ ]

[ ]

[ ]

a. Validation strategy for traning the model


We use validation_split of 0.2

Train size will be 80%, Test size is 20%


[ ]

b. Fit model to the data and analyze the results


[ ]

Loading model...


[ ]

Model's Prediction


[ ]

[ ]

[ ]

Confusion matrix


[ ]

Present loss and other metrics


[ ]

Show examples for good and bad classification with high probability and refer to the uncertain predictions


image.png


By looking at the examples, even humans might find it difficult to classify the first row. Trees look alike and it's hard to get the correct one (same for Boy-Man and Squirrel-Rabbit)

The second row doesn't have any elements that might be similar between expected and actual prediction


Compare testset result with the validation


Accuracy on testset was very similar to the validation accuracy, even a bit better

On validation: 36% | On testset: 38.7%


c. How to improve?


loss_function = sparse_categorical_crossentropy


List at least 3 ways to improve


  1. Normalize the data, using X_train.astype('float32')/255
  2. Build Conv2D layers up to size 256 (instead of 64)
  3. Use more epochs (instead of 10)
  4. Add more data to training by changing thr validation split ratio
  5. Using Augmentation

Prioritize improvements in a list


d. Implement the first 2 improvements


In the first atempt to improve we will change the following

  • Add more Conv2D layers, up to 256
  • Normalize the input data so it's values are between 0 to 1
  • Add more epochs to fit the model (we will use 50 epochs)

Build improved model


[ ]

[ ]

[ ]

[ ]

[ ]

Confusion matrix - improved


[ ]


Improve graph

As we can see, even though the improved model got better results - it's still overfitting


The red line represents the basic_model's validation status


image.png


In order to overcome the overfitting, we will try to add more samples to the training data

We will do so by setting the validation_split to 0.9


In addition, we add ImageDataGenerator for augmentation usage

Augmentation parameters

  1. rotation_range=10 ( Image will rotate at most 10 deg )
  2. width_shift_range=0.1 ( Image will shift horizontally )
  3. height_shift_range=0.1 ( Image will shift vertically )
  4. shear_range=0.01 ( Shear angle in counter-clockwise direction in degrees)
  5. zoom_range = 0.05 ( Image will be zoomed-in )
  6. horizontal_flip=True ( Allowing horizontal flip )
  7. vertical_flip=False ( Vertical flip is not allowed )

[ ]

Augmentation examples


[ ]

[ ]

[ ]

[ ]

Training the model


[ ]

[ ]

[ ]

[ ]

Predict on testset


[ ]

Confusion matrix - Improved with augmentation


[ ]

Results


Much better, we can see that both training and validation are more correlated

previous val_loss was 2.4 and val_accuracy was 0.44


----------- Training score -----------

Accuracy = 0.52 | Loss = 1.75


----------- Validation score -----------

Validation Accuracy = 0.48 | Validation Loss = 2.01


----------- Prediction score -----------

Accuracy on test set is: 48.76%


The red line represents the basic_model's validation status


image.png




3. Select a trained model architecture (in Keras or other online)


Keras pre-trained models https://keras.io/applications/


image.png


[ ]

ReduceLROnPlateau info


image.png


[ ]

[ ]

VGG19 Model


a. Change the last layer to correspond to the task


[ ]

c. Repeate 2.b


Fit the model and present metrics


[ ]

Evaluate the results


image.png


As seen in the graphs above, the model is overfitting


Load VGG model


[ ]

Confusion matrix - VGG19


[ ]

[ ]

d. Use the trained model you got in 3c as a feature extractor


[ ]

Fit RandomForest Classifier


[ ]

Results of using VGG19 model as feature extraction

The classifier is less accurate than the VGG model's accuracy ( 40% instead of 50% )


e. Preform model stacking to improve performance metrics


Improves

  1. Increase the image size ( Due to RAM limit, we will increase to 96x96 )

  2. Add layers to the model

    • Add Dropout layer to handle overfitting
    • Add BatchNormalization layers
  3. Fit using more epochs


Resize image Example


[ ]
CodeText

[ ]

[ ]

Getting a much cleaner image


Resize train and validation images


[ ]

BuildTransferModel with Stacking


[ ]

Fit model with Neptune


[ ]

Load transfer model VGG19


[ ]

Resize images


[ ]

[ ]

Confusion matrix - VGG19 with resize


[ ]

Show examples for good and bad classification with high probability and refer to the uncertain predictions


[ ]

d. Use the trained model you got in 3c as a feature extractor


[ ]

Classify using Randomforest


[ ]

Aa.*
Replace